System and method for handbag authentication

ABSTRACT

Systems and methods for authenticating handbags using a portable electronic device along with a bilinear convolutional neural network (CNN) model are described. One method includes using a portable electronic device comprising a camera, and a lens-accessory attached to the portable electronic device such that an optical feature of the lens-accessory is positioned in front of the camera. The portable electronic device acquires one or more pictures of a handbag and sends the one or more pictures to a bilinear CNN model via a network asset where an authenticity is determined. The systems and methods disclosed are capable of allowing the portable electronic device to be spaced apart from the handbag while acquiring pictures, and the lens-accessory can be between 10× and 50× magnification.

TECHNICAL FIELD

This invention relates to the image analysis of handbags byclassification with machine learning models; and more particularly, tohandbag authentication using image analysis.

BACKGROUND ART

The luxury resale market has been increasing for many years due toconsumer preferences putting an emphasis on sustainability and variety.With buying and selling of secondhand luxury goods comes an increase incounterfeiting, where a third party imitates a manufactures product withan intent to deceive and use in illegal transactions. Counterfeit goodsare typically of inferior quality and can be hard to detect by anaverage consumer. Counterfeit goods, an already major problem, will onlybecome worse as the luxury resale market continues to grow. This problemis especially prevalent in luxury handbags.

Many options have existed in the art for authentication and counterfeitidentification, such as holograms, RFID tags, and barcodes. Recently,physical unclonable functions (PUF), which is a digital signaturetypically applied to authenticate semiconductors, have also been used toauthenticate handbags.

SUMMARY OF INVENTION Technical Problem

Handbag authentication requires a person with years of training to makevisual determinations. In addition, one skilled to authenticate mustcontinue to study new releases of authentic and counterfeit handbags,especially as the difference between the two continues to diminish. Eventhose considered to be experts still require time that can slow down theauthentication process.

Current solutions such as holograms, RFID tags, and barcodes add aphysical tag to the object. This process increases time and cost of themanufacturing process. Furthermore, the physical tag can be removed,forged, or even duplicated and used to counterfeit other goods. Evenmore state-of-the art solutions including PUFs can be expensive and havebeen harder to adopt both by handbag manufacturers and resellers.

Even solutions that utilize image-analysis from machine learning rely oncustom hardware touching the handbag at a magnification greater than100×. This can slow down the authentication process and prevent fromperforming in manner similar to a factory line automation.

Solution to Problem

The advancements of machine learning technology, and especially in thefield of computer vision, have allowed computers, smartphones, and otherdevices to automatically perform actions that at one point was onlycapable of being performed by a human actor. One such process that canutilize computer vision for automatic classification is related toauthenticating luxury goods such as handbags.

The invention is directed to a method, and related systems, foridentifying counterfeit handbags. The method comprises a series ofsteps: using a portable electronic device comprising a camera, and alens-accessory attached to the portable electronic device such that anoptical feature of the lens-accessory is positioned in front of thecamera: acquiring one or more pictures of a handbag, sending the one ormore pictures to a network asset configured to execute acounterfeit-classifier model, the counterfeit-classifier model being abilinear convolutional neural network (CNN) model trained from aplurality of stored images, comparing at least a portion of each of theone or more pictures with the plurality of stored images, anddetermining if the handbag is a counterfeit based on said comparing.

Advantageous Effects of Invention

The ability to automatically authenticate a handbag with off-the-shelfhardware would result in a more efficient process than that of the priorart and could allow for factory line automation. The number oftransactions for the selling and reselling of handbags will continue toincrease and can benefit from a more efficient authentication procedure.

A person with years of experience in authenticating handbags would nolonger be required for such authentication, and the training of newemployees would be greatly reduced.

When a lens-accessory on a portable electronic device is usedconjunction with a bilinear convolutional neural network (CNN) model, anaccurate image analysis model can be trained without requiring furthercustom hardware or inefficient techniques to acquire pictures.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. The use of the same reference numbers in different instances inthe description and the figures may indicate similar or identical items.Various embodiments or examples (“examples”) of the present disclosureare disclosed in the following detailed description and the accompanyingdrawings. The drawings are not necessarily to scale. In general,operations of disclosed processes may be performed in an arbitraryorder, unless otherwise provided in the claims.

FIG. 1 is a system block diagram representation of an embodiment of ahandbag authentication system;

FIG. 2 is a flow chart representation of a bilinear CNN model;

FIG. 3 is a flow chart representation of a method of building acounterfeit-classifier model for authenticating handbags;

FIG. 4 is a flow chart representation of an example of a method of dataaugmentation;

FIG. 5 is a flow chart illustrating a method for authenticatinghandbags;

FIG. 6 is a representation of a portable electronic device and handbag;

FIG. 7 is a table of data before performing data augmentation;

FIG. 8 is a table of data after performing data augmentation; and

FIG. 9 is a table of accuracy results.

DETAILED DESCRIPTION

For purposes of explanation and not limitation, details and descriptionsof certain preferred embodiments are hereinafter provided such that onehaving ordinary skill in the art may be enabled to make and use theinvention. These details and descriptions are representative only ofcertain preferred embodiments, however, and a myriad of otherembodiments which will not be expressly described will be readilyunderstood by one having skill in the art upon a thorough review of theinstant disclosure. Accordingly, any reviewer of the instant disclosureshould interpret the scope of the invention only by the claims, as suchscope is not intended to be limited by the embodiments described andillustrated herein.

For purposes herein, “portable electronic device” means a computingdevice such as a smartphone, a tablet, or any other device havingcomputing functionality and data communication capabilities in aportable form factor;

“Lens-accessory” means an accessory configured to be attached to aportable electronic device having a camera, such that an optical featureof the lens-accessory is positioned in front of the camera. The cameraand lens-accessory may combine to have a magnification between 10× and50×;

“Counterfeit-classifier model” means a machine learning model trained toclassify one or more images of a handbag as either authentic orcounterfeit;

“Bilinear convolutional neural network (CNN)” means an imageclassification model where an image is processed through two featureextractors utilizing different CNN architecture convolutional steps,combined by a multiplication of an outer product and fed into aclassifier;

“Stored images” means a images/pictures of handbags stored on a network;

“Authentic-identifying features” means features from one or morepictures of a handbag useful to authenticate a brand and/or style of ahandbag, such features may include bumps, wrinkles, stitching, creases,color, or any other unique fabric characteristics that can be seen in amagnified image;

“Stored-image features” means features from stored images of handbagswith a known brand and/or style, such features may include bumps,wrinkles, stitching, creases, color, or any other unique fabriccharacteristics that can be seen in a magnified image;

“Outer product” means a linear algebra mathematical procedure whereinthe outer product of two vectors is a matrix. If two vectors havedimensions n and m respectively, then their outer product is an n×mmatrix. An outer product of two multidimensional arrays of numbers is atensor;

“Style” means a subclass of a brand, where a style can have distinctcharacteristics such as fabric, size, and shape that distinguishes thestyle from other styles. Multiple styles may have a common fabric.

While this disclosure is illustrative of a method and system forauthenticating handbags, the method and system can be similarly adaptedto other fashion items, such as shoes, jewelry, clothing, and the like.

In a first embodiment a method of identifying counterfeit handbags isdisclosed. The method comprises the steps of using a portable electronicdevice comprising a camera. A lens-accessory is attached to the portableelectronic device such that an optical feature of the lens-accessory ispositioned in front of the camera. The portable electronic deviceacquires one or more pictures of a handbag, sending the one or morepictures to a network asset configured to execute acounterfeit-classifier model. The counterfeit-classifier model is abilinear convolutional neural network (CNN) model trained from aplurality of stored images. Then, at least a portion of each of the oneor more pictures is compared with the plurality of stored images. Thecounterfeit-classifier model determines if the handbag is a counterfeitbased on said comparing.

In some embodiments, the portable electronic device is spaced apart fromthe handbag during said acquiring of the one or more pictures of thehandbag.

In some embodiments, the optical feature of the lens-accessory maycomprise a magnification of between 10× and 50×.

In some embodiments, said comparing the at least a portion of the one ormore pictures with the plurality of stored images comprises: for each ofthe one or more pictures, extracting a plurality ofauthentic-identifying features from the handbag for each of the one ormore pictures. The plurality of authentic identifying features is thencompared with a plurality of stored-image features from each of theplurality of stored images.

Generally, the one or more pictures are each processed through a firstfeature extractor generating a first output, a second feature extractorgenerating a second output. The first and second output are subsequentlymultiplied together by an outer product multiplication.

In some embodiments, the training of the bilinear CNN model comprises afirst feature extractor generating a plurality of first outputs from theplurality of stored images, and a second feature extractor generating aplurality of second outputs from the plurality of stored images. Anouter product multiplication is then calculated wherein the plurality ofthe first outputs is multiplied respectively with the plurality of thesecond outputs using an outer product mathematical operation.

In the first embodiment, the first feature extractor and the secondfeature extractor may each be executed on a common set of initialfeatures.

In a second embodiment, a system is disclosed. The system comprises aportable electronic device, including: a camera, a processor, and anon-transitory computer-readable medium. A lens-accessory is configuredto couple with the portable electronic device. The lens-accessoryincludes an optical feature configured for positioning in front of thecamera of the portable electronic device. The non-transitorycomputer-readable medium of the portable electronic device is configuredto store instructions that when executed by the processor cause theprocessor to perform steps comprising: acquiring one or more pictures ofa handbag, sending the one or more pictures to a network assetconfigured to execute a counterfeit-classifier model, thecounterfeit-classifier model being a bilinear convolutional neuralnetwork (CNN) model trained from a plurality of stored images, comparingat least a portion of each of the one or more pictures with theplurality of stored images, and determining if the handbag is acounterfeit based on said comparing.

In the second embodiment, the optical feature of the lens-accessory maycomprise a magnification of between 10× and 50×.

In some embodiments, the comparing the at least a portion of the one ormore pictures with the plurality of stored images includes, for each ofthe one or more pictures, extracting a plurality ofauthentic-identifying features from the handbag contained therein, andcomparing the plurality of authentic-identifying features with aplurality of stored-image features from each of the plurality of storedimages.

Generally, the one or more pictures are each processed through a firstfeature extractor and a second feature extractor in parallel. Both thefirst and second feature extractors generate a first output and a secondoutput respectively. The first and second outputs are multipliedtogether using an outer product.

In some embodiments, the training of the bilinear CNN model comprises afirst feature extractor generating a plurality of first outputs from theplurality of stored images. The training of the bilinear CNN modelfurther comprises a second feature extractor generating a plurality ofsecond outputs from the plurality of stored images. Subsequently, anouter product multiplication is determined wherein the plurality of thefirst outputs is multiplied respectively with the plurality of thesecond outputs.

In the second embodiment, the first feature extractor and the secondfeature extractor may each be executed on a common set of initialfeatures.

In a third embodiment, a non-transitory computer-readable mediumconfigured to store instructions is disclosed. The instructions, whenexecuted by one or more computers, cause the one or more computers toperform operations comprising: acquiring one or more pictures of ahandbag, sending the one or more pictures to a network asset configuredto execute a counterfeit-classifier model, the counterfeit-classifiermodel being a bilinear convolutional neural network (CNN) model trainedfrom a plurality of stored images, comparing at least a portion of eachof the one or more pictures with the plurality of stored images, anddetermining if the handbag is a counterfeit based on said comparing.

In some embodiments, said comparing the at least a portion of the one ormore pictures with the plurality of stored images comprises: for each ofthe one or more pictures, extracting a plurality ofauthentic-identifying features from the handbag contained in each of theone or more pictures, and comparing the plurality ofauthentic-identifying features with a plurality of stored-image featuresfrom each of the plurality of stored images.

Generally, the one or more pictures are each processed through a firstfeature extractor generating a first output, a second feature extractorgenerating a second output. An outer product is calculated bymultiplying the first and second outputs using an outer productoperation.

In some embodiments, the training of the bilinear CNN model includesgenerating a plurality of first outputs from the plurality of storedimages utilizing a first feature extractor. Additionally, training ofthe bilinear CNN model includes generating a plurality of second outputsfrom the plurality of stored images utilizing a second featureextractor. Once the plurality of first and second outputs are generated,the first and second outputs are combined by an outer product operation.The outer product operation must be respective, meaning only a first andsecond output from a common picture should be combined.

In the third embodiment, the first feature extractor and the secondfeature extractor may each be executed on a common set of initialfeatures.

Now to the drawings, FIG. 1 shows an embodiment of a system blockdiagram (100). The system comprises a portable electronic device (110),an authenticator interface (130), a network (120), and a server (200).The portable electronic device includes a camera (111), and attached tothe portable electronic device is a lens-accessory (112). The portableelectronic device, in conjunction with the camera and lens-accessory,acquires one or more pictures of a handbag. The one or more pictures aresent to a server (200) via a network (120). The one or more pictures areeach processed through a bilinear CNN (210) and subsequently sent to atleast one of a plurality of counterfeit-classifier models (220). Eachcounterfeit-classifier model from the plurality ofcounterfeit-classifier models is trained from a plurality of storedimages (230). After a determination is made if whether the handbag isauthentic or counterfeit, predictions are sent to the authenticatordisplay via the network.

The authenticator interface (130) can be on any device including theportable electronic device (110) or a device operated by a third party.The portable electronic device may be operated by a buyer interested inpurchasing the handbag, or alternatively by a seller interested inselling their secondhand handbag. In one embodiment, the buyerphysically receives the handbag and uses the portable electronic deviceto acquire pictures to determine if the handbag is authentic beforefinalizing the purchase. In another embodiment, the seller uses theportable electronic device to upload one or more pictures of the handbagto the server (200) to identify the handbag's authenticity. In thisembodiment, the authenticator interface would be on a device operated bythe buyer.

The portable electronic device (110) is a portable computing device suchas a smartphone, tablet, portable computer, or any other device havingcomputing functionality and data communication capabilities. Thelens-accessory (112) may include any commercially availablemagnification lens capable of magnifying an image. such as OlloClipMacro 21× Super-Fine Pro Lens(https://www.olloclip.com/products/connect-x-macro-21×-pro-lens).Preferably, the magnification would be between 10× and 50×.Alternatively, the lens-accessory can be customized in accordance withthe level of knowledge of one having skill in the art.

The one or more pictures may be sent from the portable electronic device(110) to the network (120) by an application programming interface (API)which is downloaded to the portable electronic device. The bilinear CNN(210) and the plurality of counterfeit-classifier models (220) can eachbe powered with graphic processing units (GPUs) while being deployed onthe server (200). The API can include the authenticator interface (130),and in addition may provide instructions on a number of picturesrequired along with preferred distance between the portable electronicdevice and the handbag. Generally, distances of about six to abouttwelve inches may be preferred. If a secondary classifier model iscreated and trained to classify acceptable pictures, then acceptablepictures can be automatically determined, and feedback can be given ifan insufficient number of acceptable pictures are provided.

The bilinear CNN (210) comprises a set of processes to extract featuresfrom the one or more pictures prior to sending to at least one from theplurality of counterfeit-classifier models. The set of processes includeextracting features with two feature extractors, sum-pooling, and outerproduct. Further discussion of the bilinear CNN can be found in FIG. 2 .

At least one counterfeit-classifier model from the plurality ofcounterfeit-classifier models (220) receives features from an output ofthe bilinear CNN (210), including extracted features, and processes thefeatures through a classifier to make a prediction. The prediction iswhether the handbag is authentic or counterfeit. The plurality ofcounterfeit-classifier models is trained from a plurality of storedimages (230). Preferably, each trained model from the plurality ofcounterfeit-classifier models is trained on one type of fabric. Forexample, Monogram is a fabric used by Louis Vuitton on over one-hundreddifferent styles of handbags. If a fabric used has common features amongdifferent styles of handbags, then stored images comprising the fabriccan be consolidated and used to train a model. The model is then capableto infer if a particular fabric from a picture is a counterfeit or isgenuine.

In one embodiment, characteristics of the handbag are identified priorto authentic classification. Based on identified characteristics, suchas brand/style or fabric, the one or more pictures can be sent to acorrectly trained model. The correctly trained model is trained fromstored images comprising fabric common to the identifiedcharacteristics. Determination of the characteristics can be performedautomatically by a handbag classifier trained for identifying handbagcharacteristics. Alternatively, the determination of characteristics canbe performed manually by someone having skill in the art andsubsequently inputting the characteristics into the API of the portableelectronic device (110). In another embodiment, the handbagcharacteristics are ignored and instead the one or more pictures areclassified by each of the plurality of counterfeit-classifier models,whereby models which output a classification below a preset probabilityare disregarded and only a counterfeit-classifier model that produces aprediction above the preset probability is selected.

Alterative to having a plurality of counterfeit-classifier models, amulti-class model can be trained on a plurality of fabrics to make aprediction of authenticity. The multi-class model would comprise aplurality of classifications, each classification comprising a fabrictype and either an authentic or counterfeit label.

Referring to FIG. 2 , a flow chart representation of a bilinear CNNmodel (210) is shown. In the model, an image (211) is fed into a firstfeature extractor (212 a) and a second feature extractor (212 b). Thefirst and second feature extractors can each be one of a variety of CNNarchitecture models truncated at a convolutional layer of the respectiveCNN architecture model. Examples include M-Net comprising 14 layers ofconvolution, and D-Net comprising 30 layers of convolution. After animage is processed through the first and second feature extractors, afirst output (213 a) and second output (213 b) are respectively created.The first and second outputs and then multiplied together by an outerproduct (214), after which, sum-pooling (215) is then executed toaggregate bilinear features across the image, resulting in a bilinearvector. The bilinear vector is then fed in a classifier (216).

The classifier (216) can either be a linear classifier or nonlinearclassifier. Linear classifiers include support vector machines (SVM),logistic regression, or perceptron. Nonlinear classifiers includek-nearest neighbors (kNN) and kernel SVM. Logistic regression can beadvantageous due to outputting a probability of a maximum likelihoodestimation. However, it can be appreciated that alternative classifierscan also be used, and may also comprise a plurality of fully-connectedlayers.

Referring to FIG. 3 , a flow chart representation of a method ofbuilding a counterfeit-classifier model (300) for authenticatinghandbags is shown. The method of building a counterfeit-classifier modelcomprises the steps data acquisition (310), data augmentation (320),training (330), and testing (340).

Data acquisition (310) may be conducted by human curators. Each of thehuman curators can use portable electronic devices to collect picturesfrom both authentic handbags and counterfeit handbags. Each of theportable electronic devices are capable of acquiring pictures withmagnification up to 50×. It is preferred if each of the portableelectronic devices were similar in performance to provide consistency ofpictures acquired. When acquiring pictures from a handbag with aparticular fabric, multiple images should be captured of various partsthe particular fabric. Given the possibility of having a limited numberof counterfeit handbags, it is important to ensure that there is asufficient number of counterfeit handbags saved for testing aftertraining is performed. As for the authentic handbags, a similar numbercan be set aside for testing, or alternatively, newly arrived handbagsfrom the handbag's manufacture may be used.

As has been described already, a fabric that is used on multiple handbagstyles can be consolidated as long as features of the fabric are commonamong the multiple handbag styles. Factors that should be consideredinclude whether a common fabric is sourced for all related styles, andwhether each handbag style undergoes similar pre- and post-processingmanufacturing steps. Common features can be determined manually by onehaving skill in the art of counterfeit handbags by sampling fabrics ofdifferent styles and subsequently compared. Alternatively, handbagmanufacturers can be contacted to confirm common fabric is used amongdifferent styles.

Data augmentation (320) may be needed to address class imbalance. Theclass imbalance, where data for counterfeit handbags is significantlyoutweighed by data for authentic handbags, can affect quality andreliability of results. Classifiers tend to become biased towardsclassifying authentic handbags and do not perform as well classifyingcounterfeit handbags. To rectify a class imbalance, pictures ofcounterfeit handbags can be augmented, or perturbed, to increase thedata for counterfeit handbags. For each picture of a handbag fabric,which alternatively can be described as a training example, a randomperturbation process is selected and applied the picture. The randomperturbation process is performed with a uniformly random parameter set.For example, perturbation processes that may be considered includecontrast, scaling, gaussian noise, and image rotation. A uniformlyrandom parameter set of the perturbation process of image rotation couldcomprise −30 degrees, −20 degrees, −10 degrees, +10 degrees, +20degrees, and +30 degrees. Data augmentation by perturbation is describedmore in FIG. 4 .

Training (340) is conducted by a bilinear convolutional neural network(CNN) as described in FIG. 2 . The bilinear CNN can model local pairwisefeature interactions in a translationally invariant manner. Furthermore,the bilinear CNN can simplify gradient computation which allows end toend training of both first and second feature extractors using onlylabels of an image. CNN architectures used as the first and secondfeature extractors can be pre-trained on an image dataset, such asImageNet, which is then subsequently followed by fine-tuning specific tohandbags. Outputs of both the first and second feature extractors aremultiplied using an outer product operation at each location of theimage and pooled to obtain a bilinear vector. The bilinear vector isthen passed through a classifier to obtain predictions. A preferablemethod of training a counterfeit-classifier model includes training aseparate model for each fabric. Alternatively, a single model can betrained on all fabric types with only pictures of authentic bags. If aprobability output from a softmax layer is below a threshold amount,then the handbag is classified as counterfeit. The single model approachis based on logic that a handbag that doesn't match closely with any ofa plurality of authentic bags is predicted as a counterfeit.

Testing (340) is the step that proceeds training (330). Testing isperformed with holdover data which was not used during testing. It canbe advantageous that pictures used during training and testing eachcomprise a mixture of old and new to smooth out any procurement ormanufacturing changes that inevitably can occur over time. To ensurecounterfeit handbags are identified, a threshold of the softmax layercan be deliberately chosen to catch all counterfeits whilemisclassifying a small portion of authentic handbags.

Referring to FIG. 4 , a flow chart representation of an example of amethod of data augmentation (320) is shown. The method comprises thesteps:

Step 1: Determine a desired increase per counterfeit training example(321). One example of a method to determine the desired increase is asfollows: calculate a difference of a number of authentic and counterfeittraining examples and divide by the number of counterfeit trainingexamples. The method described herein will equalize the number ofcounterfeit training examples with the number of authentic trainingexamples.

Step 2: Select one counterfeit training example from the plurality ofcounterfeit examples (322).

Step 3: Randomly select a perturbation process from a group ofperturbation processes (323). Performing a selection randomly is used toprevent any bias from a particular perturbation process.

Step 4: Process the counterfeit training example through the randomlyselected perturbation process to create a perturbed image (324).

Step 5: Consolidate the perturbed image with any previously perturbedimages from a common counterfeit training example (325).

Once all perturbed images from a counterfeit training example areconsolidated into a total number of perturbed images, the total numberof perturbed images are compared to the desired increase determined inStep 1 (321). If the total number of perturbed images is less than thedesired amount, the method returns to Step 3 (323) where anotherperturbation process is randomly selected and performed. If the totalnumber of perturbed images is greater than or equal to the desiredamount, then the training example is removed from the plurality ofcounterfeit training examples. An assessment of the plurality ofcounterfeit training examples is then conducted. If there are nocounterfeit training examples remaining, then the data augmentation iscomplete. Otherwise, the method returns to Step 2 (322) where a newcounterfeit training example is selected and processed similarly.

Alternative approaches can be made from the flowchart described in FIG.4 . They may include randomly selecting only one perturbation processonce, and performing said perturbation process multiple times for acounterfeit training example. Another approach involves randomlyselecting a plurality of perturbation processes for each counterfeittraining example and performing each of the plurality of perturbationprocesses only one time.

Referring to FIG. 5 , a flow chart illustrating a method forauthenticating handbags (400) is shown. The method for authenticatinghandbags comprises the steps: acquiring one or more pictures of ahandbag (410), sending the one or more pictures to a network assetconfigured to execute a counterfeit-classifier model (420), comparing atleast a portion of each of the one or more pictures with a plurality ofstored images (430), and determining if the handbag is a counterfeitbased on said comparing (440).

The acquiring one or more pictures of a handbag (410) step comprises aportable electronic device having a camera, and a lens-accessory coupledto the portable electronic device. The lens-accessory may have amagnification level between 10× and 50×. The portable electronic devicerequires a distance from the handbag close enough such that the one ormore pictures acquired embody features that will subsequently be fed ina bilinear CNN. Said features may include bumps, wrinkles, stitching,creases, color, or any other unique fabric characteristics that can beseen in a magnified image. However, positioning the camera such that thecamera is touching the handbag is not necessary. This allows for a moreefficient process of acquiring pictures. The efficient process could beleveraged, for example, by creating a factory line authenticationprocess wherein multiple handbags can be consecutively authenticatedusing an assembly line model.

The step of sending the or more pictures to a network asset configuredto execute a counterfeit-classifier model (420) is generally executednext. It is preferable to include, either before or with the one or morepictures, a brand and style which can be inputted by an operator of theportable electronic device. Including the brand and style has severalpurposes, including verifying that said brand and style has been trainedin the counterfeit-classifier model. If the counterfeit-classifier modelhas not been trained on the brand and style, the operator of theportable electronic device can be notified. Another purpose of includingthe brand and style prior to or with the one or more pictures relates towhen there is a plurality of counterfeit-classifier models each trainedon a separate fabric. The brand and style can be used to send the one ormore pictures to an appropriate model from the plurality ofcounterfeit-classifier models.

After the one or more pictures are sent, the step of comparing at leasta portion of each of the one or more pictures with a plurality of storedimages (430) follows. More specifically, the one or more images areprocessed through a bilinear CNN and subsequent classifier, saidsubsequent classifier is trained from the plurality of stored images.Training the classifier with the plurality of the stored images includesextracting features from the plurality of stored images. The featuresfrom the plurality of stored images are ultimately compared withfeatures extracted from the one or more pictures.

The final step of the method authenticating a handbag comprises the stepof determining if the handbag is a counterfeit based on said comparing(440). A threshold of a softmax output can be deliberately chosen tocatch all counterfeit handbags while misclassifying a small portion ofauthentic handbags. If a counterfeit classification is charactered as apositive, then it can be appreciated by one having skill in the art thatthe threshold of the softmax output maximizes recall while making asacrifice in precision. This is preferred due to an importance ofcatching counterfeits.

If a plurality of pictures is sent to a counterfeit-classifier model, itis possible, though unlikely, that some of the plurality of pictures maybe predicted as authentic while others will be predicted as counterfeit,despite the plurality of pictures comprising a common handbag.Procedures such as a majority vote can be employed to determine a finalpredicted classification. A percentage needed to generate the finalpredicted classification can be customized wherein an amount lower thanthe percentage would return a counterfeit classification or noclassification at all.

Referring to FIG. 6 , a representation of a portable electronic deviceand handbag (500) is shown. The representation comprises a portableelectronic device (510) and a handbag (520). The portable electronicdevice includes a lens-accessory (512), wherein the lens-accessory andthe portable electronic device, when used together, is capable ofacquiring pictures having a magnification between 10× and 50×. Theportable electronic device is capable of acquiring one or more photos ofthe handbag while being spaced apart therewith. The handbag comprises afabric (521) which includes features that can make the handbagdistinctive from other handbags.

Referring to FIG. 7 , a table of data before performing dataaugmentation is shown. The table of data before data augmentationdemonstrates a problem of class imbalance between authentic images andcounterfeit images. For example, Louis Vuitton having the Monogramfabric comprises 8021 authentic images and only 2008 counterfeit. Tocorrect the class imbalance, a method, such as described in FIG. 4 , maybe used to create more counterfeit images through perturbation.

Referring to FIG. 8 , a table of data after performing data augmentationis shown. This table shows the quantities of both authentic images andcounterfeit images. Number of authentic images remains the same, andnumber of counterfeit images have been increased via data augmentationto match the number of authentic images. It is not necessary that thenumber of counterfeit images must match the number of authentic images,and one having skill in the art will appreciate that a different numberof counterfeit images can still solve class imbalance problems.

Referring to FIG. 9 , a table of accuracy results is shown. The tableexhibits testing results from three different fabrics, namely Monogram,DamierEbene, and Caviar. A model trained with training data of theMonogram fabric was tested with 562 authentic images and 242 counterfeitimages, producing a 98% and 100% accuracy respectively. Similar resultsare also shown for the DamierEbene and Caviar. Increasing or decreasingfalse negatives and false positives can be achieved by modifying athreshold of a softmax output. It will be appreciated by one havingskill in the art that to decrease false negatives or false positiveswill likely increase the other.

What is claimed is:
 1. A method of authenticating a handbag comprising:receiving a picture of the handbag having a plurality ofauthentic-identifying features; receiving one or more characteristics ofthe handbag; determining a fabric associated with the handbag based onthe one or more characteristics; selecting from among a plurality ofcounterfeit-classifier models each trained on one type of fabric, acounterfeit-classifier model trained on the fabric associated with thehandbag; processing the picture through the counterfeit-classifier modeltrained on the fabric associated with the handbag; and determiningauthentication of said handbag.
 2. The method of claim 1, wherein theone or more characteristics comprise brand, style, the fabric, or acombination thereof.
 3. The method of claim 1, wherein thecounterfeit-classifier model comprises a bilinear convolutional neuralnetwork (CNN) model trained from plurality of images.
 4. The method ofclaim 1, wherein the processing the picture through thecounterfeit-classifier model trained on the fabric associated with thehandbag comprises: generating a first output from a first featureextracted from a first feature extractor; generating a second outputfrom a second feature extracted from a second feature extractor; andmultiplying the first and second outputs by an outer product.
 5. Themethod of claim 1, wherein the processing the picture through thecounterfeit-classifier model trained on the fabric associated withhandbag comprises: extracting the plurality of authentic-identifyingfeatures of the pictures; comparing the plurality ofauthentic-identifying features of the picture with a plurality ofauthentic-identifying features associated with thecounterfeit-classifier model.
 6. A system for authenticating a handbagcomprising: a processor and a non-transitory computer-readable medium,the non-transitory computer-readable medium of the electronic devicebeing configured to store instructions, the instructions when executedby the processor cause the processor to perform steps comprising:receiving a picture of the handbag having a plurality ofauthentic-identifying features; receiving one or more characteristics ofthe handbag; determining a fabric associated with the handbag based onthe one or more characteristics; selecting from among a plurality ofcounterfeit-classifier models each trained on one type of fabric, acounterfeit-classifier model trained on the fabric associated with thehandbag; processing the picture through the counterfeit-classifier modeltrained on the fabric associated with the handbag; and determiningauthentication of said handbag.
 7. The system of claim 6, wherein theone or more characteristics comprise brand, style, the fabric, or acombination thereof.
 8. The system of claim 6, wherein thecounterfeit-classifier model comprises a bilinear convolutional neuralnetwork (CNN) model trained from plurality of images.
 9. The system ofclaim 6, wherein the processing the picture through thecounterfeit-classifier model trained on the fabric associated with thehandbag comprises: generating a first output from a first featureextracted from a first feature extractor; generating a second outputfrom a second feature extracted from a second feature extractor; andmultiplying the first and second outputs by an outer product.
 10. Thesystem of claim 6, wherein the processing the picture through thecounterfeit-classifier model trained on the fabric associated withhandbag comprises: extracting the plurality of authentic-identifyingfeatures of the pictures; comparing the plurality ofauthentic-identifying features of the picture with a plurality ofauthentic-identifying features associated with thecounterfeit-classifier model.
 11. A non-transitory computer-readablemedium configured to store instructions, the instructions when executedby one or more computers, cause the one or more computers to performoperations comprising: receiving a picture of a handbag having aplurality of authentic-identifying features; receiving one or morecharacteristics of the handbag; determining a fabric associated with thehandbag based on the one or more characteristics; selecting from among aplurality of counterfeit-classifier models each trained on one type offabric, a counterfeit-classifier model trained on the fabric associatedwith the handbag; processing the picture through thecounterfeit-classifier model trained on the fabric associated with thehandbag; and determining authentication of said handbag.
 12. Thenon-transitory computer-readable medium of claim 11, wherein the one ormore characteristics comprise brand, style, the fabric, or a combinationthereof.
 13. The non-transitory computer-readable medium of claim 11,wherein the counterfeit-classifier model comprises a bilinearconvolutional neural network (CNN) model trained from plurality ofimages.
 14. The non-transitory computer-readable medium of claim 11,wherein the processing the picture through the counterfeit-classifiermodel trained on the fabric associated with the handbag comprises:generating a first output from a first feature extracted from a firstfeature extractor; generating a second output from a second featureextracted from a second feature extractor; and multiplying the first andsecond outputs by an outer product.
 15. The non-transitorycomputer-readable medium of claim 11, wherein the processing the picturethrough the counterfeit-classifier model trained on the fabricassociated with handbag comprises: extracting the plurality ofauthentic-identifying features of the pictures; comparing the pluralityof authentic-identifying features of the picture with a plurality ofauthentic-identifying features associated with thecounterfeit-classifier model.